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Abstract 

Reporting and interpreting (corrected) effect size indices help readers of scientific reports more 
fully understand the importance of those results, rather than just the likelihood of results 
assuming the null hypothesis is true. This paper explores the frequency of effect reporting and 
interpretation in an analysis of quantitative studies in Communication Education, the flagship 
instructional communication journal of the National Communication Association. Over half of 
the studies (57%) reported effect sizes and the majority of those (88%) interpreted these effects. 
The authors discuss implications concerning effect reporting and interpretation and include good 
examples of effect interpretation. 
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In Search of OZ: Effect Size Reporting and Interpretation in Communication Research 

Girl with ruby slippers travels with cohorts in search of passage back to Kansas. Girl is told that a being 
named OZ is all-powerful and all-knowing and will be able to provide meaningful answers to inquiries. 
Girl and companions— person of little scruples, person without a heart, and person of questionable 
courage — ultimately discover OZ is no more than a facade merely appearing to have the final word on 
such quandaries and does not, in the end, provide complete assistance in helping girl get back to Kansas. 

This oversimplification of the Wizard of Oz is symbolic of the growing need in research 
circles to add stronger support to statistical significance testing by reporting and interpreting 
effect size indices. For too long, researchers have boldly proclaimed that the importance of their 
study exist with the probability value (p-value). By providing effect sizes, researchers are 
strengthening their claims with both magnitude of effect and practical significance, both of 
which a p-value cannot and does not speak to. Reporting and interpreting effect size indices help 
readers of scientific reports more fully understand the importance of those results, rather than 
just the likelihood of results assuming the null hypothesis is true. In short, and in keeping with 
the metaphor, p-values do not completely help researchers get to Kansas! If the p-value is OZ, 
then effect size reporting and interpretation may just be the ruby slippers needed to help 
researchers facilitate a better understanding of what their results mean. 

Of course, effect sizes are not magical, but useful for evaluating how noteworthy the 
research findings are. Nevertheless, the movement toward reporting effects in addition to or 
instead of p-values has developed some inertia. As a reflection of this trend, the following known 
journals now require that an effect index be reported along with p-values (Vacha-Haase, 2001): 
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Career Development Quarterly 
Contemporary Educational Psychology 
Educational and Psychological Measurement 
Exceptional Children 
Journal of Agricultural Education 
Journal of Applied Psychology 
Journal of Community Psychology 
Journal of Consulting & Clinical Psychology 
Journal of Counseling and Development 
Journal of Early Intervention 

Journal of Educational and Psychological Consultation 
Journal of Experimental Education 
Journal of Learning Disabilities 
Language Learning 

Measurement and Evaluation in Counseling and Development 
The Professional Educator 
Research in the Schools 



Purpose 

Several articles have reviewed the current state of effect reporting (cf. Henson & Smith; 
Vacha-Haase, Nilson, Reetz, Lance, & Thompson, 2000). These empirical reviews were 
conducted largely for educational and psychological journals and suggest effect sizes are still 
often omitted from research articles. However, few recent reviews exist for the communication 
research literature (Chase & Simpson, 1979; Holley & Barker, 1979; Kutzer & Sodt, 1973) and 
therefore it is unclear the degree that authors are currently including effects in communication 
articles. Therefore, this paper examined reporting practices within quantitative research reports 
published in the National Communication Association’s flagship journal on instructional 
communication, Communication Education. An explanation of effect size reporting and 
interpretation is provided as background for the present study and to establish the need for 
reporting effect sizes in research as set out by the American Psychological Association (APA) 
Publication Manual (APA, 2001) and current trends. The authors present and explain findings 
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from a review of quantitative studies published in Communication Education during the year 

2000. 

Effect Size Reporting 

Effect size, in its simplest form, describes the degree of difference in the dependent 
measure accounted for by the presence or absence of the independent measure (Holley & Barker, 
1979), or the degree of relationship between variables. As used by some, an effect size augments 
the statistical significance test by determining how strong or meaningful, subsequent to a 
researcher interpretation, a treatment, measure, or variable’s relationship with the dependent 
variable. Without assessing effect sizes, a researcher knows only that a difference or relationship 
exists, but not how important, useful, or big that difference is. Thompson (1994) — and to date, 
the 5 th edition of the APA Publication Manual (APA, 2001) — noted that semantically stating just 
that results are “significant” implies that such are especially important, noteworthy, or 
“practically” significant. Statistical significance, however, does not necessarily equate practical 
significance. By using effect size estimates, researchers can probe their data to determine the 
practical significance (importance) of their results (Cohen, 1994; Roberts & Henson, in press; 
Thompson, in press). 

Given the importance of reporting effects, Snyder and Lawson (1993) argued that 
researchers should understand both the categories of effect estimates and the details inherent to 
calculating effects in order to correctly use such in their analysis and reporting. A brief review 
of these concepts follows. 

Mean Difference vs. Variance-Accounted-For Effects 

Two general categories of effects exist: those that involve directly examining differences 
between means and those that involve proportions of variance (e.g. variance-accounted-for 
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estimates). Examples of effects concerning mean differences include mean difference indices 
and standardized differences between means (such as Cohen’s d). These indices help explain the 
degree of group differences that exist. 

In comparison, variance-accounted-for estimates describe the amount of variability 
explained or predicted by the independent variable in the dependent variable. Examples of these 
indices include eta , partial eta , omega , epsilon , R , partial R , Ezekiel formula, Herzberg 
formula, and the Lord formula (Snyder & Lawson, 1993). While different, both are related in the 
general linear model in that Cohen’s d can be converted to an r 2 -type effect, due to the 
correlational nature of all data analytic procedures (Cohen, 1994). 

Uncorrected vs. Corrected Effects 

Furthermore, two types of variance-accounted-for effect size estimates exist. Also called 
biased magnitude of association estimates, uncorrected effect estimates can be explained as the 
ratio of explained variance to total variance. Examples of these include eta (as used in ANOVA) 
and R 2 (as used in regression). These estimates are often referred to as biased because they 
capitalize on sampling error and tend to overestimate the population effect or the effect in future 
samples (Roberts & Henson, in press; Snyder & Lawson, 1993; Yin & Fan, 2001). 

Corrected effect estimates (unbiased magnitude of association estimates), conversely, 
attempt to statistically correct for the positive bias that occurs in uncorrected estimates. Using 
corrected effect estimates allow the researcher to better estimate the true population effect — 
which bolsters the generalizability of one’s results to future samples — and hence, are 
recommended for use in analysis and in the interpretation of results (Snyder & Lawson, 1993; 
Thompson & Kieffer, 2000). Various formulas exist for computing corrected effect sizes. 
Examples of these estimates include adjusted R 2 (Ezekiel formula often used in regression) and 
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omega 2 (ANOVA). Adjusted R 2 is automatically calculated for regression procedures by most 
statistical packages even when not requested, however the actual calculation is: 

Adj. R 2 = R 2 - ((1 -R 2 )*(v/n-v - 1)), 

where R is the sample effect, v is the number of predictors, and n is the sample size. 

The omega 2 estimates can be expressed as: 

omega = (SSbetween- ( k — 1) x MSwithin ) • (SStotal + MSwithin ), 
where k is the number of groups (Hinkle & Wiersma, 1998). 

Effect Size Interpretation 

Along with reporting effect sizes, researchers are also encouraged to interpret these in 
their discussion of results. Failure to discuss the meaning of one’s effect does not help in 
determining whether or not the difference detected is important, useful, or practical (Thompson, 
1996). A significant issue of concern surrounding the interpretation of effect sizes is the current 
position stated in the 5 th edition of the APA Publication Manual (APA, 2001). Additional issues 
concern interpreting effect size in lieu of sample size and a possible error in SPSS for Windows 
with eta 2 . 

APA Publication Manual 

Most social science journals require articles to conform to the APA Publication Manual 
for style. The 4 th edition of the Publication Manual addressed the lack of specificity innate to p- 
values by “encouraging” researchers to either report effect sizes or provide the reader the means 
by which to calculate the effect size (test statistic and sample size/degrees of freedom) in their 
research results. However, few researchers heeded the call as evidenced by reviews of effect 
reporting (Henson & Smith, 2000; Vacha-Haase et al., 2000). The debate over statistical 
significance testing prompted the APA to convene a formal discussion (APA Task Force on 
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Statistical Inference) to review the many issues researchers and methodologists were discussing 
at conferences, journals, and meetings regarding statistical significance testing. Harlow, Mulaik, 
and Steiger (1997) provide a balanced discussion of the debate. In its report (Wilkinson & APA 
Task Force on Statistical Inference, 1999), the Task Force set out several suggestions for 
reporting statistically significant tests and argued that effect sizes should be reported and 
interpreted. Despite the APA Task Force’s mandate for effect size reporting and interpretation, 
the 5 th edition of the APA Publication Manual (2001) stopped short of issuing an explicit 
requirement. Nevertheless, the 5 th edition Publication Manual suggests there is seldom an 
occasion when effects should not be reported. According to the Publication Manual (APA, 
2001 ): 

For the reader to fully understand the importance of your findings, it is almost always 
necessary to include some index of effect size of strength of relationship in your Results 
section. . .The principle to be followed, however, is to provide the reader not only with 
information about statistical significance but also with enough information to assess the 
magnitude of the observed effect of relationship, (pp. 25-26) 

Thompson and Snyder (1997) further suggested that, 

. ..explicitly and reflectively linking research results in a given study to the effect sizes in 
previous studies is also a vehicle for evaluating result repliciability. This can be done 
prospectively by formulating null hypotheses incorporating specific parameter 
expectations derived from previous research, as against the contemporary practice of 
always testing hypotheses of no difference or of no relationship (i.e., what Cohen, 1994, 
described as “nil” hypothesis testing), (p. 80) 
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The APA Task Force (Wilkinson & APA Task Force on Statistical Inference, 1999) concurred, 
“We must stress again that reporting and interpreting effect sizes in the context of previously 
reported effects is essential to good research. It enables readers to evaluate the stability of results 
across samples, designs, and analyses” (p. 599). 

Sample Size 

One issue that the 4 th edition Publication Manual spoke more directly to (than the 5 th 
edition does) is the impact of sample size on statistical significance testing. According to the 4 th 
edition Publication Manual (APA, 1994), “the ways of reporting probability values do not reflect 
the importance or magnitude of an effect or strength of a relationship because the probability 
values depend on sample size... [therefore] you are encouraged to provide effect size 
information” (p. 1 8). Put another way, statistically significant results are bound to occur if the 
sample size is large enough (Henson & Smith, 2000). When the sample size is small, samples 
contain more sampling error and “flukiness” (Thompson, 1996, p. 27) and will typically yield 
larger Ecalculated values. Large samples, conversely, will yield smaller Ecalculated values 
likely giving way to statistically significant results. Thompson (1996) referred to statistical 
significance testing as merely a test of “researcher endurance” (p. 27) in that results will be 
statistically significant, albeit not guaranteed to be practically significant, when one has rounded 
up enough participants in his/her sample. 

Therefore, statistical significance tests should be interpreted in light of the sample size 
used to compute the E-value. Further, such tests should be viewed in light of the observed effect 
that is being tested. Small (perhaps immaterial) effects can be statistically significant at large 
sample sizes; large (perhaps noteworthy) effects may not be statistically significant if the study 
suffers from a small sample size and lack of power. 
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‘‘What If ’ Analyses 

In an attempt to clarify the conundrum of sample size influence on statistical significant 
testing, Thompson and Kieffer (2000) offered the “what if’ analysis to aid researchers in 
interpreting their results through the lens of sample size and effect size. In brief, “what if’ 
analyses assesses at what sample size the study results would either become statistically 
significant or not, keeping (corrected) effect size constant. The benefit of such an approach is 
that it reveals whether or not a slight (or dramatic) difference in sample size would still yield a 
statistically significant result or cause the loss of statistical significance. As Snyder and Lawson 
(1993) noted, such a post hoc power analysis can be an important “result interpretation aid” (p. 
33) and can prevent researchers with large samples from overinterpreting their small effects once 
they see that with a difference of a few participants in their sample, their results would no longer 
be statistically significant. The same is true for researchers with large effects, in that they can 
demonstrate a statistically significant result would also have occurred with a smaller sample size 
(Thompson & Kieffer, 2000). 

Possible Concerns 

A possible error in the interpretation of eta 2 has been noted by Levine and Hullett (2000) 
in SPSS for Windows. They found that partial eta 2 was reported as eta 2 on the printouts even 
though it was correctly noted to be partial eta 2 in the software documentation. Such an error may 
cause an overinterpretation of effect and could prove problematic for researchers using it to 
measure effect in a factorial way ANOVA or in meta-analyses where many effect sizes across 
studies are assessed (Levine & Hullett, 2000). 
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Effect Size Reporting and Interpretation in Communication Education 
As early as the 1970’s, the debate about “meaningful and trivial” results was occurring in 
the communication discipline (Chase & Simpson, 1979; Holley & Barker, 1979; Katzer & Sodt, 
1973). As mentioned before, journals that require such reporting (Communication Monographs 
and Human Communication Research) are among the few in the National Communication 
Association’s arsenal of seven internationally recognized publications. Katzer and Sodt (1979) 
conducted a similar analysis of effect size reporting and interpretation in articles published in the 
1971 and 1972 volumes of the Journal of Communication. The authors were concerned with 
four issues in their analyses: (a) whether journal articles reported effect sizes, (b) if effect sizes 
were not reported, did sufficient information exist to compute the effect size, c) whether the 
author(s) interpreted the effect size (rather than just reporting it), and (d) if effect sizes were 
confused with probability values (alpha or p). They found that only one researcher (out of 22) 
actually reported (and interpreted!) the effect size in their results. In lieu of these results, Katz 
and Sodt (1973) issued a staunch directive to communication researchers in terms of reporting 
effect size estimates in their studies: “If researchers feel compelled to affect the tentative model 
of communication, then it is even more important for OES [observed effect size] measures to be 
routinely reported (p. 256). 

Method 

The present paper presents a review similar to that of Katzer and Sodt (1973). However, 
we focused our attention specifically on effect reporting and interpretation. Kirk (1996) noted 
higher frequency of effect reporting in regression research likely due to the fact that most 
statistical software packages routinely provide variance-accounted-for effects in their output. Our 
attention concerned two primary issues: (a) what articles reported (variance-accounted-for) effect 
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sizes for statistically significant results, and (b) did the author(s) interpret the effect size in their 
discussion of findings. 

To explore these issues, we analyzed articles published in the 2000 volume of 
Communication Education. Excluding book reviews, reports, and essays, 14 articles that used 
statistical analysis were published during that year. For each article, we identified the major 
statistical test, the number of statistically significant results, whether authors reported effects 
and, if so, which effect size indices were used, and whether authors interpreted their effects. 

Results and Discussion 

In the fourteen articles studied, 420 statistical tests were employed. Of these, 216 yielded 
statistically significant results. A discussion of effect reporting and interpretation practices in the 
fourteen articles follows. 

Effects Reported 

Eight articles (57%) reported effect sizes for their statistically significant results. This 
results stands in stark contrast to Katzer and Sodt’s (1973) finding of 4.5 percent and is 
incredibly high given that Communication Education does not specifically require that authors 
report (or interpret) effects in their submission policy, rather just that manuscripts must abide by 
the current APA Publication Manual. The effect sizes used most were eta (50%) and R (30%), 
with omega 2 and Cohen’s d used in the remaining studies. As noted earlier, both eta 2 and R 2 are 
uncorrected effect estimates and tend to overestimate population effects or the effect in future 
samples. As most researchers are concerned about the generalizability of their findings, use of 
corrected effects may be more applicable. 
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Effects Interpreted 

Of the eight articles that reported effects, seven (88%) interpreted the effect size in their 
discussion. Each discussed the implications of the effects differently, but all emphasized how 
important and thus practically significant the results were, beyond the statistically significant 
p-value result. For example, in a study comparing techniques for helping students manage high 
communication apprehension (CA), Dwyer (2000) argued that while skills training via a public 
speaking course did help high CAs reduce their anxiety level, the Multidimensional Model had a 
“significantly greater impact”. The authors described the Model as a “highly effective way to 
help students reduce CA” in interpersonal and public contexts and that it “extends the 
possibilities for intervention and instruction” in a communication course (p. 78). Another good 
example of effect interpretation is in a study that explores the relationships between interactive 
television instructors’ perceptions of students’ nonverbal responsiveness and the influence of 
these perceptions on distance teaching. Mottet (2000) found that interactive television instructors 
perceived more of their students nonverbal responsive behaviors, formed more positive student 
impressions, evaluated themselves as being more effective with their teaching style and in the 
relationships formed with students when in the two-way audio/video condition that in the two- 
way audio/one-way video. In terms of effect, Mottet (2000) argued: 

. . .the largest impact was that of the nonverbal responsiveness measure measures and the 
teacher-student interpersonal relationship measure. The interactive capabilities accounted 
for 13% of the variance in the higher-interference nonverbal responsive measure and 14% 
of the variance in the low-interference nonverbal audible measure. Additionally, the 
interactive capabilities of the technology accounted for 13% of the variance in the 
teacher-student interpersonal relationship measure, (p. 160) 
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Finally, King, Young, and Behnke (2000) explained a low effect size (eta 2 = .085) in a study 
comparing immediate versus delayed feedback in improving public speaking through the 
ComET system. They contended that while the Hawthorne effect may have been at work in the 
groups that received immediate feedback in having a more positive attitude (compared to the 
delayed feedback group), they stated “it is hopeful and noteworthy that such systems may be 
used to supplement traditional instruction with no apparent loss of positive affect nor increase in 
performance anxiety” (p. 372). 

Conclusion 

Communication researchers are encouraged to report and interpret (corrected) effect 
indices in their research. Such reporting and interpretation aids both the reader and the researcher 
understands more fully the results at hand. Results become “more understandable, and [such 
reporting and interpretation] facilitates informed reader judgment regarding the trustworthiness 
and noteworthiness of the results” (Thompson, 1999, p. 332). Such credibility is important as 
the National Communication Association puts forth various initiatives to increase both the 
visibility and acceptance of communication research among public and private granting agencies, 
within the databases and evaluation programs of the National Research Council and the 
Department of Education, and while influencing policy makers who design laws that govern 
communication practices (Applegate, 2001, p. 3). 

Many social scientists call on textbook authors, journal editors, and publication manuals 
to lead the way to proper reporting and interpretation of data-analytic techniques (Hyde, 2001; 
Vaccha-Haase, 2001). Clearly, Communication Education boasts a healthy composite of effect 
reporting and interpretation in the issues reviewed, but that number could be 100% if the 
editorial policy would require such reporting and explanation of effects rather than relying on a 
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precarious interpretation of the APA Publication Manual guidelines. In the 4 th edition 
Publication Manual, the mere “encouraging” of effect size reporting sends a contradictory 
message that some reporting requirements count while others do not (Thompson, 1999). 
Fortunately, the 5 th edition Publication Manual is more direct. As social science research 
continues to move toward effect reporting and interpretation, communication researchers would 
do well to embrace this methodological reform. Perhaps Communication Education could 
consider an editorial policy to help facilitate this outcome. 
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